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Response to Arguments 

Applicant's arguments filed 3/13/2008 have been fully considered but they are not 
persuasive. 

In response to Applicant's argument (on page 9 in first paragraph) that the reference 
Sawhney does not state the video information is projected on to such models, Contrarily, 
Sawhney teaches clearly in fig. 8d, see model red lines. On pages 9-10, Applicant argues that 
Sawhney does not describe, placing a surface that corresponds to a moving region in a three 
dimensional model, Examiner believes Sawhney e.g., in fig. 9 illustrates a moving region (a 
moving aerial) in a 3-D model e.g., using model lines in fig. 8d. 

In response to Applicant's argument on page 10 in second paragraph the reference's 
technique described a moving object uses stereo cameras, and this is in sharp contrast with the 
claimed invention, which can be employed using a single camera. Examiner's reply: in light of 
the specification e.g. in fig. 10 illustrates at least three cameras. 

Examiner's notes: Applicant my point out where is the limitation of using only a single 
camera in the claims? 

In response to Applicant's argument on page 1 1 in first paragraph, regarding Arpa the 
reference that used in previous office action dated 5/4/2005, It is noted the current reference is 
"Sawhney" based on office action dated 1 1/13/2007. 

Examiner's suggestion: Applicant is encouraged to schedule an interview. 

Further down on the same page, Applicant argues the reference does not indicate that the 
threshold is histogram based or that a noise filter is used. Examiner points out to e.g., page 164 at 
the right col. teaches the absolute value at each point can then be computed, and the result can be 
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thresholded in order to highlight intensity or feature differences between the current video image 
and the reference image, on page 160 in fig. 3 clearly illustrates final rendered composite that has 
been filtered the video pixels. 

On page 12 Applicant indicated, is not understood the Examiner interpretation that the 
noise filter may be considered as a virtual camera view in fig. 3 that shows just the video pixels; 
wherein identifying a region in motion in real time further comprises estimating the background 
image by modeling the background image as a temporal pixel average of five recent image 
frames in the real-time video imagery information (e.g., on page 164 at right col. teaches the 
reference background image needs to be constantly updated during the day to reflect changing 
ambient illumination. Alternatively only moving objects may be detected by comparing current 
image with an image taken a few seconds before. The 2D moving object detection can be done in 
real time, also see page 162 left column at third paragraph). On the other hand, comparing fig. 15 
of the current invention with fig. 10 of the reference: in fig. 15 steps 1500, 1510, 1520, and 1530 
illustrate what is shown the left side of fig. 10, and fig. 10 illustrates the validated foreground 
object in step 1550 in the middle of fig. 10. 

Examiner's notes: Applicant's arguments do not clearly point out the patentable novelty 
which he or she thinks the claims present in view of the state of the art disclosed by the 
references cited or the objections made. Further, they do not show how the amendments avoid 
such references or objections. 

Based on mentioned reasons above, the previous rejection is still maintained. 
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Claim Rejections - 35 USC §103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 25, 29-31, 33, 37-39, and 41-50 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over H. S. Sawhney, et al. with title of "Video Flashlights - Real Time Rendering 
of Multiple Videos for Immersive Model Visualization", hereinafter Sawhney, see cited PTO- 
1449 dated 8/6/07. 
Claim 29. 

A method comprising: Sawhney teaches obtaining a three dimensional model of a three 
dimensional environment (e.g., under section 2 on page 158 in the third paragraph), the three 
dimensional model generated from range sensor (e.g., left col. line 5 on page 165), information 
representing a height field for the three dimensional environment (e.g., a range finder device); 
identifying in real time a region in motion (e.g., the video sequence shown moving objects in fig. 
10) with respect to a background image (e.g., in fig. 10 see the background image) in real-time 
video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model (e.g., in fig. 10 see 3D 
change image), the background image comprising a single distribution background dynamically 
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modeled from a time average of the real-time video imagery information (e.g., in fig. 10 
illustrates the background image and a video sequence, on page 167); 

Sawhney is silenced explicitly specify (see underlined, below) placing a surface that 
corresponds to the moving region in the three dimensional model (e.g., in fig. 10 illustrates 
moving objects), wherein placing the two dimensional surface comprises: casting a ray 
(Examiner's note: casting rays are: cast and traced in groups based on some geometric 
constraints, and each ray is traced separately, so that every point (usually a pixel) on the display 
is traced by one ray.) from an optical center , corresponding to the real-time video imagery 
information, to a bottom point of the moving region in an image plane in the three dimensional 
model; and determining a position, an orientation and a size of the two dimensional surface 
based on the ray, a ground plane in the three dimensional model, and the moving region (e.g., in 
fig. 10 illustrates a 3D model, and a moving region). Projecting the real-time video imagery 
information onto the three dimensional model, including the surface, based on the position and 
orientation information (e.g., in fig. 10 illustrates moving objects); and visualizing the three 
dimensional model with the projected real-time video imagery (e.g., in fig. 10 illustrates moving 
objects); wherein identifying a region in motion in real time comprises subtracting the 
background image from the real-time video imagery information (e.g., in fig. 10 illustrates on the 
right and left sides of the figure), identifying a foreground object in the subtracted real-time 
video imagery information, validating the foreground object by correlation matching between 
identified objects in neighboring image frames, and outputting the validated foreground object 
(e.g., in fig. 10 illustrates outputting the foreground object vs the background image at the right 
side of the figure); 
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However, Examiner believes Sawhney teaches the optical center in fig. 4 that 
corresponds depicting occlusion handling for rendering a video flashlight camera and the model 
from a virtual viewpoint, because it would have been obvious to an ordinary person skill in the 
art to recognize that Sawhney compares the two alpha values for all points in order to determine 
which points are illuminated by the camera, and obviously one may refer cameras in fig. 4 of 
Sawhney as the optical center in the claimed invention. 

Claims 25, and 46. 

Sawhney teaches the method of claim 29, wherein the surface comprises a two dimensional 
surface (e.g., in fig. 5 illustrates 2D surfaces). 

Claims 30, 38, and 47. 

The method of claim 29, further comprising tracking the position (e.g., see moving objects 
represented as 3D masks. Only moving people or cars are detected) and orientation information 
of the at least one image sensor in the environment with respect to the three dimensional model 
in real-time (e.g., in fig. 10 illustrates a 3D model, see in 3Dchange image that detects moving 
people and cars). Claims 38 and 47 are rejected with similar reasons as set forth in claim 30, 
above. 

Claims 31, 39, and 48. 

The method of claim 30, wherein obtaining a three dimensional model of a three dimensional 
environment comprises generating the three dimensional model of the three dimensional 
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environment (e.g., in fig. 8 illustrates a 3D model, see (d)). Claims 48, and 39 are rejected with 
similar reasons as set forth in claim 31, above. 

Claim 33. 

The system of claim 37, wherein the surface comprises a two dimensional surface (e.g., under 
section 2 at third paragraph teaches the 2D surface). 

Claim 37. 

Claim 37 is rejected with similar reasons as set forth in claim 29, above. 

Regarding new claims 41-44, wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time video imagery information using a 
histogram-based threshold and a noise filter (e.g., on page 164 at the right col. teaches the 
absolute value at each point can then be computed, and the result can be thresholded in order to 
highlight intensity or feature differences between the current video image and the reference 
image, on page 160 in fig. 3 clearly illustrates final rendered composite that has been filtered the 
video pixels, and Sawhney does not disclose "a noise filter" Examiner interpretation: the noise 
filter may be considered as a virtual camera view in fig. 3 (must be seen in color image) that 
shows just the video pixels); wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the background image as a temporal 
pixel average of five recent image frames in the real-time video imagery information (e.g., on 
page 164 at right col. teaches the reference background image needs to be constantly updated 
during the day to reflect changing ambient illumination. Alternatively only moving objects may 
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be detected by comparing current image with an image taken a few seconds before. The 2D 
moving object detection can be done in real time). 

Sawhney does not explicitly specify ( see the underlined sections ) identifying a region in 
motion in real time by modeling the background image as a temporal pixel average of five recent 
image frames in the real time video imagery. However, Examiner takes an official notice for an 
average of five recent frames, because Sawhney on page 161 under "A pseudo-code for the 
rendering algorithm .." teaches as one of the parameters that a user may be indicated a number 
for number of frames "frame number", and that number may be 5 image frames, also on page 
164 at the bottom of first paragraph left column i.e. comparing current image with an image 
taken a few seconds before. The 2D moving object detection can be done in real time. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to substitute applicant 's described structure, material, or acts for that described in the Sawhney's 
system/method, in order to take advantage of the Video Flashlight system that is the real-time 
rendering of multiple live videos overlaid precisely on a 3D model. 

Claim 45 is rejected with similar reasons as set forth in claim 29, above, except the 
preamble of claim 45 contains "a machine-readable medium . . . .", see rejection under 112. 

Claims 49-50 are rejected with similar reasons as set forth in claims 41-42, above. 
Claim Rejections - 35 USC § 112 

The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

Claims 45-50 are rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite for 
failing to particularly point out and distinctly claim the subject matter which applicant regards as 
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the invention, the preamble of claims 45-50 contain "machine-readable medium ..." that renders 
the claims indefinite, because the machine-readable medium is not well established whether is a 
camera, a computer or an optical device. The limitation of "medium" is not explicitly defined 
whether is a storage device, or a signal, or etc. 

Claims 46-50 are contained similar limitations as set forth in claim 45, therefore they are 
rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite. 

Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JAVID A. AMINI whose telephone number is (571)272-7654. 
The examiner can normally be reached on 8-4pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on 571-272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Javid A Amini 
Primary Examiner 
Art Unit 2628 

/Javid A Amini/ 

Primary Examiner, Art Unit 2628 



